build: gyp node_base #23439

refack · 2018-10-12T03:08:07Z

Motivation:

node_base as an independant target, that we can run just to make sure the code in /src/ compiles.
Easy to compose with code_cache_stub.cc to build an mkcodecache exe. Then later relink with the generated `code_cache_gen.cc.
Easier to run "whole program" analysis
Move to /src/node_base.gypi to make node.gyp easier to read.
Maybe solve AIX specific linking issue

side-fixes:

eliminate rename_node_bin_win by naming the node lib file on Windows libnode.lib
enable PCH for the node sources
explicitly define Environment::context() so it can safely reside in two translation units. (otherwise undefined behavior emerges)
minor .h cleanup to ensure node_lib builds correctly.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
documentation is changed or added
commit message follows commit guidelines

nodejs-github-bot · 2018-10-12T03:08:08Z

@refack build started: https://ci.nodejs.org/blue/organizations/jenkins/node-test-pull-request-lite-pipeline/detail/node-test-pull-request-lite-pipeline/1211/pipeline

refack · 2018-10-12T03:17:23Z

Sanity CI: ~~https://ci.nodejs.org/job/node-test-pull-request/17747/~~
https://ci.nodejs.org/job/node-test-pull-request/17748/
/CC @nodejs/build-files @nodejs/gyp @nodejs/platform-windows

refack · 2018-10-13T23:05:29Z

CI: ~~https://ci.nodejs.org/job/node-test-pull-request/17838/~~
https://ci.nodejs.org/job/node-test-commit/22309/

If I could get some review, this refactoring makes it for a much better experience in MSVS. It's now possible to only compile node_base and get a result within a minute, if you changes compile of not.

refack · 2018-10-15T23:11:09Z

New CI: https://ci.nodejs.org/job/node-test-commit/22334/

refack · 2018-10-23T02:10:56Z

On Windows and SmartOS Getting:

20:39:21 gyp ERR! UNCAUGHT EXCEPTION 
20:39:21 gyp ERR! stack ReferenceError: DTRACE_NET_STREAM_END is not defined
20:39:21 gyp ERR! stack     at Socket.end (net.js:550:3)
20:39:21 gyp ERR! stack     at findVS2017 (c:\workspace\node-test-binary-windows\deps\npm\node_modules\node-gyp\lib\find-vs2017.js:43:15)
20:39:21 gyp ERR! stack     at c:\workspace\node-test-binary-windows\deps\npm\node_modules\node-gyp\lib\configure.js:94:9
20:39:21 gyp ERR! stack     at c:\workspace\node-test-binary-windows\deps\npm\node_modules\mkdirp\index.js:30:20
20:39:21 gyp ERR! stack     at FSReqCallback.oncomplete (fs.js:148:20)
20:39:21 gyp ERR! System Windows_NT 10.0.10240

Which is a linking issue...

refack · 2018-10-23T02:12:17Z

On macOS:

20:18:59   rm -f /Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1011/out/Release/libnode.a && ./gyp-mac-tool filter-libtool libtool -Wl,-force_load,/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1011/out/Release/libnode_base.a -static -o /Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1011/out/Release/libnode.a /Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1011/out/Release/obj.target/node_lib/src/node_code_cache_stub.o /Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1011/out/Release/obj.target/node_lib/src/node_dtrace.o /Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1011/out/Release/obj.target/node_lib/gen/node_javascript.o
20:18:59 error: /Library/Developer/CommandLineTools/usr/bin/libtool: unknown option character `W' in: -Wl,-force_load,/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1011/out/Release/libnode_base.a

Which means you can't "force load" with libtool.

richardlau · 2018-10-23T02:14:06Z

Maybe solve AIX specific linking issue

@refack Any more details on what the linking issue was/is?

refack · 2018-10-23T02:38:18Z

Any more details on what the linking issue was/is?

There's a bunch of code in node.gyp that specific to AIX

node/node.gyp

Lines 925 to 972 in f1b9546

    
           'conditions': [ 
        
             [ 'OS=="aix" and node_shared=="true"', { 
        
               'variables': {'real_os_name': '<!(uname -s)',}, 
        
               'targets': [ 
        
                 { 
        
                   'target_name': 'node_aix_shared', 
        
                   'type': 'shared_library', 
        
                   'product_name': '<(node_core_target_name)', 
        
                   'ldflags': [ '--shared' ], 
        
                   'product_extension': '<(shlib_suffix)', 
        
                   'conditions': [ 
        
                     ['target_arch=="ppc64"', { 
        
                       'ldflags': [ 
        
                         '-Wl,-blibpath:/usr/lib:/lib:' 
        
                           '/opt/freeware/lib/pthread/ppc64' 
        
                       ], 
        
                     }], 
        
                     ['target_arch=="ppc"', { 
        
                       'ldflags': [ 
        
                         '-Wl,-blibpath:/usr/lib:/lib:/opt/freeware/lib/pthread' 
        
                       ], 
        
                     }], 
        
                     ['"<(real_os_name)"=="OS400"', { 
        
                       'ldflags': [ 
        
                         '-Wl,-blibpath:/QOpenSys/pkgs/lib:/QOpenSys/usr/lib', 
        
                         '-Wl,-bbigtoc', 
        
                         '-Wl,-brtl', 
        
                       ], 
        
                     }], 
        
                   ], 
        
                   'includes': [ 
        
                     'node.gypi' 
        
                   ], 
        
                   'dependencies': [ '<(node_lib_target_name)' ], 
        
                   'include_dirs': [ 
        
                     'src', 
        
                     'deps/v8/include', 
        
                   ], 
        
                   'sources': [ 
        
                     '<@(library_files)', 
        
                     'common.gypi', 
        
                   ], 
        
                   'direct_dependent_settings': { 
        
                     'ldflags': [ '-Wl,-brtl' ], 
        
                   }, 
        
                 }, 
        
               ] 
        
             }], # end aix section

I would like to fold it into the new node_base target...

refack · 2018-10-29T01:59:41Z

CI: https://ci.nodejs.org/job/node-test-pull-request/18191/

refack · 2018-10-29T18:43:40Z

CI: https://ci.nodejs.org/job/node-test-commit/22730/

refack · 2018-10-30T16:10:53Z

Resume CI: https://ci.nodejs.org/job/node-test-commit/22761/

richardlau · 2018-10-30T23:53:27Z

@refack I'm assuming the presence of the dontland! commit means this is still in progress and added the label.

refack · 2018-10-31T00:23:09Z

Yeah. I thought I had this figured out, but it keeps surprising me with new edge cases...
(the dontland! is to overcome compiler OOM issues of Windows 🤦‍♂️)

If you have the time I'd love some review on the first few commits, they are pretty complete:

Create node_base - 3a7fdce
Add more pre-compiled headers on Windows -27d769b
Only do LTCG for the node exe - 3521f8a
Refactor ETW and DTRACE gyp - 2ce719d

refack · 2018-10-31T00:24:20Z

Good news, I can now do a full (cold cache) windows build in 20% less time, and with clcache and warm cache, full windows build in < 4 minutes.

~~Smoke test CI: https://ci.nodejs.org/job/node-test-pull-request/18263/~~
Smoke test CI: https://ci.nodejs.org/job/node-test-pull-request/18264/

refack · 2018-11-01T22:25:13Z

CI: https://ci.nodejs.org/job/node-test-pull-request/18333/

* remove unused includes * make writing content-stable as to not trigger rebuilding

* move `arraysize` to util.h to reduce the need for `node_intrnals.h` * make sure options_parser singletons init in order * deoptimize NodeTraceWriter

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs related to general changes in the lib or src directory. labels Oct 12, 2018

refack requested review from addaleax, bnoordhuis and joyeecheung October 12, 2018 03:08

refack self-assigned this Oct 12, 2018

refack added windows Issues and PRs related to the Windows platform. build Issues and PRs related to build files or the CI. wip Issues and PRs that are still a work in progress. labels Oct 12, 2018

refack force-pushed the create-node_base branch 3 times, most recently from db8bdb6 to 682c932 Compare October 13, 2018 23:02

refack removed the wip Issues and PRs that are still a work in progress. label Oct 13, 2018

refack force-pushed the create-node_base branch from 1fd5cfb to dd230eb Compare October 15, 2018 22:50

refack mentioned this pull request Oct 15, 2018

src: add public wrapper for Environment::GetCurrent #23676

Merged

2 tasks

refack mentioned this pull request Oct 16, 2018

Build node with GN #21410

Closed

refack force-pushed the create-node_base branch from 4055df7 to 2706a2b Compare October 29, 2018 01:58

refack force-pushed the create-node_base branch 2 times, most recently from 0e09b56 to 1cdc26f Compare October 29, 2018 02:19

refack force-pushed the create-node_base branch from cc9f80c to d821a49 Compare October 30, 2018 20:38

richardlau added the wip Issues and PRs that are still a work in progress. label Oct 30, 2018

refack force-pushed the create-node_base branch 2 times, most recently from a874629 to f084e58 Compare October 31, 2018 16:36

refack added 9 commits November 4, 2018 22:01

tools: tweak js2c.py

543c1d8

* remove unused includes * make writing content-stable as to not trigger rebuilding

src: fixup for embedding - reduce dependecy

0a17e27

* move `arraysize` to util.h to reduce the need for `node_intrnals.h` * make sure options_parser singletons init in order * deoptimize NodeTraceWriter

build: create node_base gyp target

cfa64dc

build,src: add PCH to node_base

3e230c3

deps,v8: fix gypfile bug

db60be8

build: better deafults for V8

6643da4

build,src,inspector: break includes cycle

5dbf8d8

build: refacor dtrace & etw build

b22cc31

dontland! windows workaround

290a1b3

refack force-pushed the create-node_base branch from 6c24c3b to 290a1b3 Compare November 5, 2018 03:08

refack mentioned this pull request Nov 23, 2018

build: spawn make test-ci with -j1 #23733

Merged

3 tasks

refack closed this Dec 21, 2018

refack deleted the create-node_base branch December 21, 2018 01:32

refack removed their assignment Mar 11, 2019

Uh oh!

build: gyp node_base #23439

build: gyp node_base #23439

Uh oh!

Conversation

refack commented Oct 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

nodejs-github-bot commented Oct 12, 2018

Uh oh!

refack commented Oct 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

refack commented Oct 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

refack commented Oct 15, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

refack commented Oct 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

refack commented Oct 23, 2018

Uh oh!

richardlau commented Oct 23, 2018

Uh oh!

refack commented Oct 23, 2018

Uh oh!

refack commented Oct 29, 2018

Uh oh!

refack commented Oct 29, 2018

Uh oh!

refack commented Oct 30, 2018

Uh oh!

richardlau commented Oct 30, 2018

Uh oh!

refack commented Oct 31, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

refack commented Oct 31, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

refack commented Nov 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

refack commented Oct 12, 2018 •

edited

Loading

refack commented Oct 12, 2018 •

edited

Loading

refack commented Oct 13, 2018 •

edited

Loading

refack commented Oct 15, 2018 •

edited

Loading

refack commented Oct 23, 2018 •

edited

Loading

refack commented Oct 31, 2018 •

edited

Loading

refack commented Oct 31, 2018 •

edited

Loading

refack commented Nov 1, 2018 •

edited

Loading